Efficient and sparse neural networks by pruning weights in a multiobjective learning approach

Overparameterization and overfitting are common concerns when designing training deep neural networks, that often counteracted by pruning regularization strategies. However, these strategies remain secondary to most learning approaches suffer from time computational intensive procedures. We suggest a multiobjective perspective on the of networks treating its prediction accuracy network complexity as two individual objective functions in biobjective optimization problem. As showcase example, we use cross entropy measure while adopting an l1-penalty function assess total cost (or complexity) parameters. The latter is combined with intra-training approach reinforces reduction requires only marginal extra cost. From optimization, this truly large-scale compare different paradigms: On one hand, adopt scalarization-based transforms problem into series weighted-sum scalarizations. other hand implement stochastic multi-gradient descent algorithms generate single Pareto optimal solution without requiring or using preference information. In first case, favorable knee solutions identified repeated runs adaptively selected scalarization Numerical results exemplary convolutional confirm large reductions negligible loss possible.